‘More than just numbers on a page?’ A qualitative exploration of the use of data collection and feedback in youth mental health services

Objectives This study aimed to explore current data collection and feedback practice, in the form of monitoring and evaluation, among youth mental health (YMH) services and healthcare commissioners; and to identify barriers and enablers to this practice. Design Qualitative semi-structured interviews were conducted via Zoom videoconferencing software. Data collection and analysis were informed by the Theoretical Domains Framework (TDF). Data were deductively coded to the 14 domains of the TDF and inductively coded to generate belief statements. Setting Healthcare commissioning organisations and YMH services in Australia. Participants Twenty staff from healthcare commissioning organisations and twenty staff from YMH services. Results The umbrella behaviour ‘monitoring and evaluation’ (ME) can be sub-divided into 10 specific sub-behaviours (e.g. planning and preparing, providing technical assistance, reviewing and interpreting data) performed by healthcare commissioners and YMH services. One hundred belief statements relating to individual, social, or environmental barriers and enablers were generated. Both participant groups articulated a desire to improve the use of ME for quality improvement and had particular interest in understanding the experiences of young people and families. Identified enablers included services and commissioners working in partnership, data literacy (including the ability to set appropriate performance indicators), relational skills, and provision of meaningful feedback. Barriers included data that did not adequately depict service performance, problems with data processes and tools, and the significant burden that data collection places on YMH services with the limited resources they have to do it. Conclusions Importantly, this study illustrated that the use of ME could be improved. YMH services, healthcare commissioners should collaborate on ME plans and meaningfully involve young people and families where possible. Targets, performance indicators, and outcome measures should explicitly link to YMH service quality improvement; and ME plans should include qualitative data. Streamlined data collection processes will reduce unnecessary burden, and YMH services should have the capability to interrogate their own data and generate reports. Healthcare commissioners should also ensure that they provide meaningful feedback to their commissioned services, and local and national organisations collecting youth mental health data should facilitate the sharing of this data. The results of the study should be used to design theory-informed strategies to improve ME use.


Evidence of strong beliefs impacting on behaviour
The reality is in the Australian commissioning environment the politics of commissioning, the relationships with government, and the proximity of government to their commissioners means that we can make a decision to say, well, the K10s are getting worse. You're not achieving any outcome here. But government will say, you will continue to fund them anyway. That makes a bit of a joke of the whole use of analytics to drive improvement process.
(Participant 17) Commissioned services have limited funds to allocate to monitoring and evaluation 4/4

Evidence of strong beliefs impacting on behaviour, belief shared with youth mental health services
The difficulty is the service providers, I really feel for them, because they live and breathe off the contracts they get. We don't build in specifically into the contracts that evaluation time and resource and that generally speaking they're expected just to do that as part of the work that we commission. But our commissioning dollars are reasonably tight, so when we write the contracts we deliberately write the contracts for the majority of that money to be spent on direct service delivery. (

Conflicting beliefs present
We work together pretty well as a team. Our program coordinators try and link in together, so that we're learning off each other.

(Participant 18) -----Conflicting belief-------------------------------------------------------
The hAPI platform itself can actually generate reports or collect a lot of that information. Particularly the new system is quite sophisticated in the sense that it can just extract an actual report for you in real time. (Participant 14) My organisation is supportive of the use of monitoring and evaluation information  ----------------------------------------------------- We actually don't have an enormous amount of use for the information that comes back from [Headspace] head office, partly because the information's de-identified. In the process, even though the information comes from our service and from data that we input, they believe that they're restricted in terms of giving that information back to us. So they'll give us back information, but it's only according to client number, not according to the client's name and that creates a lot of problems for us in terms of working out basically, in terms of using the information essentially for service improvement.

(Participant 27) -----Conflicting belief-------------------------------------------------------Then every quarter that information is collected and provided
back to the service about how many sessions, how many young people were seen and what were they presenting with, what was their age. So, we do get a lot of information from that platform about the young people and the demographics and I guess the story of our headspace centre and primary. I find that really helpful.

Social influences
The PHN actively supports us 34/14

Evidence of strong beliefs impacting on behaviour, conflicting beliefs present
…the PHNs that I find helpful are the ones who are willing to work in partnership, rather than seeing it as a -there are commissioners who have described themselves as like an ATM, you complete the transaction and we give you the money. Whereas others are more likely to work in partnership, so really collaborative kind of decision making.